Shared Memory Fences In our last adventure, dri3k first steps, one of the future work items was to deal with synchronization between the direct rendering application and the X server. DRI2 handles this by performing a round trip each time the application starts using a buffer that was being used by the X server. As DRI3 manages buffer allocation within the application, there s really no reason to talk to the server, so this implicit serialization point just isn t available to us. As I mentioned last time, James Jones and Aaron Plattner added an explicit GPU serialization system to the Sync extension. These SyncFences serializing rendering between two X clients, but within the server there are hooks provided for the driver to use hardware-specific serialization primitives. The existing Linux DRM interfaces queue rendering to the GPU in the order requests are made to the kernel, so we don t need the ability to serialize within the GPU, we just need to serialize requests to the kernel. Simple CPU-based serialization gating access to the GPU will suffice here, at least for the current set of drivers. GPU access which is not mediated by the kernel will presumably require serialization that involves the GPU itself. We ll leave that for a future adventure though; the goal today is to build something that works with the current Linux DRM interfaces. SyncFence Semantics The semantics required by SyncFences is for multiple clients to block on a fence which a single client then triggers. All of the blocked clients start executing requests immediately after the trigger fires. There are four basic operations on SyncFences:

• Trigger. Mark the fence as ready and wake up all waiting clients
• Await. Block until the fence is ready.
• Query. Retrieve the current state of the fence.
• Reset. Unset the fence; future Await requests will block.

SyncFences are the same as Events as provided by Python and other systems. Of course all of the names have been changed to keep things interesting. I ll call them Fences here, to be consistent with the current X usage. Using Pthread Primitives One fact about pthreads that I recently learned is that the synchronization primitives (mutexes, barriers and semaphores) are actually supposed to work across process boundaries, if those objects are in shared memory mapped by each process. That seemed like a great simplification for this project; allocate a page of shared memory, map into the X server and direct rendering application and use the existing pthreads APIs. Alas, the pthread objects are architecture specific. I m pretty sure that when that spec was written, no-one ever thought of running multiple architectures within the same memory space. I went and looked at the code to check, and found that each of these objects has a different size and structure on x86 and x86_64 architectures. That makes it pretty hard to use this API within X as we often have both 32- and 64- bit applications talking to the same (presumably 64-bit) X server. As a last resort, I read through a bunch of articles on using futexes directly within applications and decided that it was probably possible to implement what I needed in an architecture-independent fashion. Futexes Linux Futexes live in this strange limbo of being a not-quite-public kernel interface. Glibc uses them internally to implement locking primitives, but it doesn t export any direct interface to the system call. Certainly they re easy to use incorrectly, but it s unusual in the Linux space to have our fundamental tools locked away for our own safety . Fortunately, we can still get at futexes by creating our own syscall wrappers.

static inline long sys_futex(void *addr1, int op, int val1,
                 struct timespec *timeout, void *addr2, int val3)
 
    return syscall(SYS_futex, addr1, op, val1, timeout, addr2, val3);
 

For this little exercise, I created two simple wrappers, one to block on a futex:

static inline int futex_wait(int32_t *addr, int32_t value)  
    return sys_futex(addr, FUTEX_WAIT, value, NULL, NULL, 0);
 

and one to wake up all futex waiters:

static inline int futex_wake(int32_t *addr)  
    return sys_futex(addr, FUTEX_WAKE, MAXINT, NULL, NULL, 0);
 

Atomic Memory Operations I need atomic memory operations to keep separate cores from seeing different values of the fence value, GCC defines a few such primitives and I picked _syncboolcompareandswap and _syncvalcompareandswap. I also need fetch and store operations that the compiler won t shuffle around:

#define barrier() __asm__ __volatile__("": : :"memory")
static inline void atomic_store(int32_t *f, int32_t v)
 
    barrier();
    *f = v;
    barrier();
 
static inline int32_t atomic_fetch(int32_t *a)
 
    int32_t v;
    barrier();
    v = *a;
    barrier();
    return v;
 

If your machine doesn t make these two operations atomic, then you would redefine these as needed. Futex-based Fences These wake-all semantics of Fences greatly simplify reasoning about the operation as there s no need to ensure that only a single thread runs past Await, the only requirement is that no threads pass the Await operation until the fence is triggered. A Fence is defined by a single 32-bit integer which can take one of three values:

• 0 - The fence is not triggered, and there are no waiters.
• 1 - The fence is triggered (there can be no waiters at this point).
• -1 - The fence is not triggered, and there are waiters (one or more).

With those, I built the fence operations as follows. Here s Await:

int fence_await(int32_t *f)
 
    while (__sync_val_compare_and_swap(f, 0, -1) != 1)  
        if (futex_wait(f, -1))  
            if (errno != EWOULDBLOCK)
                return -1;
         
     
    return 0;
 

The basic requirement that the thread not run until the fence is triggered is met by fetching the current value of the fence and comparing it with 1. Until it is signaled, that comparison will return false. The compareandswap operation makes sure the fence is -1 before the thread calls futex_wait, either it was already -1 in the case where there were other waiters, or it was 0 before and is now -1 in the case where there were no waiters before. This needs to be an atomic operation so that the fence value will be seen as -1 by the trigger operation if there are any threads in the syscall. The futex_wait call will return once the value is no longer -1, it also ensures that the thread won t block if the trigger occurs between the swap and the syscall. Here s the Trigger function:

int fence_trigger(int32_t *f)
 
    if (__sync_val_compare_and_swap(f, 0, 1) == -1)  
        atomic_store(f, 1);
        if (futex_wake(f) < 0)
            return -1;
     
    return 0;
 

The atomic compareandswap operation will make sure that no Await thread swaps the 0 for a -1 while the trigger is changing the value from 0 to 1; either the Await switches from 0 to -1 or the Trigger switches from 0 to 1. If the value before the compareandswap was -1, then there may be threads waiting on the Fence. An atomic store, constructed with two memory barriers and a regular store operation, to mark the Fence triggered is followed by the futex_wake call to unblock all Awaiting threads. The Query function is just an atomic fetch:

int fence_query(int32_t *f)
 
    return atomic_fetch(f) == 1;
 

Reset requires a compareandswap so that it doesn t disturb things if the fence has already been reset and there are threads waiting on it:

void fence_reset(int32_t *f)
 
    __sync_bool_compare_and_swap(f, 1, 0);
 

A Request for Review Ok, so we ve all tried to create synchronization primitives only to find that our obvious implementations were full of holes. I d love to hear from you if you ve identified any problems in the above code, or if you can figure out how to use the existing glibc primitives for this operation.

DRI3K First Steps Here s an update on DRI3000. I ll start by describing what I ve managed to get working and then summarize discussions that happened on the xorg-devel mailing list. Private Back Buffers One of the big goals for DRI3000 is to finish the job of moving buffer management out of the X server and into applications. The only thing still allocated by DRI2 in the X server are back buffers; everything else moved to the client side. Yes, I know, this breaks the GLX requirement for sharing buffers between applications, but we just don t care anymore. As a quick hack, I figured out how to do this with DRI2 today allocate our back buffers separately by creating X pixmaps for them, and then using the existing DRI2GetBuffersWithFormat request to get a GEM handle for them. Of course, now that all I ve got is a pixmap, I can t use the existing DRI2 swap buffer support, so for now I m just using CopyArea to get stuff on the screen. But, that works fine, as long as you don t care about synchronization. Handling Window Resize The biggest pain in DRI2 has been dealing with window resize. When the window resizes in the X server, a new back buffer is allocated and the old one discarded. An event is delivered to invalidate the old back buffer, but anything done between the time the back buffer is discarded and when the application responds to the event is lost. You can easily see this with any GL application today resize the window and you ll see occasional black frames. By allocating the back buffer in the application, the application handles the resize within GL; at some point in the rendering process the resize is discovered, and GL creates a new buffer, copies the existing data over, and continues rendering. So, the rendered data are never lost, and every frame gets displayed on the screen (although, perhaps at the wrong size). The puzzle here was how to tell that the window was resized. Ideally, we d have the application tell us when it received the X configure notify event and was drawing the frame at the new size. We thought of a cute hack that might do this; track GL calls to change the viewport and make sure the back buffer could hold the viewport contents. In theory, the application would receive the X configure notify event, change the viewport and render at the new size. Tracking the viewport settings for an entire frame and constructing their bounding box should describe the size of the window; at least it should describe the intended size of the window. There s at least one serious problem with this plan applications may well call glClear before calling glViewport, and as glClear does not use the current viewport, instead clearing the whole window, we couldn t use the viewport as an indication of the current window size. However, what this exercise did lead us to realize was that we don t care what size the window actually is, we only care what size the application thinks it is. More accurately, the GL library just needs to be aware of any window configuration changes before the application, so that it will construct a buffer that is not older than the application knowledge of the window size. I came up with two possible mechanisms here; the first was to construct a shared memory block between application and X server where the X server would store window configuration changes and signal the application by incrementing a sequence number in the shared page; the GL library would simply look at the sequence number and reallocate buffers when it changed. The problem with the shared memory plan was that it wouldn t work across the network, and we have a future project in mind to replace GLX indirect rendering with local direct rendering and PutImage which still needs accurate window size tracking. More about that project in a future post though X Events to the Rescue So, I decided to just have the X server send me events when the window size changed. I could simply use the existing X configure notify events, but that would require a huge infrastructure change in the application so that my GL library could get those events and have the application also see them. Not knowing what the application is up to, we d have to track every ChangeWindowAttributes call and make sure the event_mask included the right bits. Ick. Fortunately, there s another reason to use a new event we need more information than is provided in the ConfigureNotify event; as you know, the Swap extension wants to have applications draw their content within a larger buffer that can have the window decorations placed around it to avoid a copy from back buffer to window buffer. So, our new ConfigureNotify event would also contain that information. Making sure that ConfigureNotify event is delivered before the core ConfigureNotify event ensures that the GL library should always be able to know about window size changes before the application. Splitting the XCB Event Stream Ok, so I ve got these new events coming from the X server. I don t want the application to have to receive them and hand them down to the GL library; that would mean changing every application on the planet, something which doesn t seem very likely at all. Xlib does this kind of thing by allowing applications to stick themselves into the middle of the event processing code with a callback to filter out the events they re interested in before they hit the main event queue. That s how DRI2 captures Invalidate events, and it works , but using callbacks from the middle of the X event processing code creates all kinds of locking nightmares. As discussed above, I don t care when GL sees the configure events, as long as it gets them before the application finds about about the window size change. So, we don t need to synchronously handle these events, we just need to be able to know they ve arrived and then handle them on the next call to a GL drawing function. What I ve created as a prototype is the ability to identify specific events and place them in a separate event queue, and when events are placed in that event queue, to bump a sequence number so that the application can quickly identify that there s something to process. Making the Event Mask Per-API Instead of Per-Client The problem described above about using the core ConfigureNotify events made me think about how to manage multiple APIs all wanting to track window configuration. For core events, the selection of which events to receive is all based on the client; each client has a single event mask, and each client receives one copy of each event. Monolithic applications work fine with this model; there s one place in the application selecting for events and one place processing them. However, modern applications end up using different APIs for 3D, 2D and media. Getting those libraries to cooperate and use a common API for event management seems pretty intractable. Making the X server treat each API as a separate entity seemed a whole lot easier; if two APIs want events, just have them register separately and deliver two events flagged for the separate APIs. So, the new DRI3 configure notify events are created with their own XID to identify the client-side owner of the event. Within the X server, this required a tiny change; we already needed to allocate an XID for each event selection so that it could be automatically cleaned up when the client exited, so the only change was to use the one provided by the client instead of allocating one in the server. On the wire, the event includes this new XID so that the library can use it to sort out which event queue to stick the event in using the new XCB event stream splitting code. Current Status The above section describes the work that I ve got running; with it, I can run GL applications and have them correctly track window size changes without losing a frame. It s all available on the dri3 branches of my various repositories for xcb proto, libxcb, dri3proto and the X server. Future Directions The first obvious change needed is to move the configuration events from the DRI3 extension to the as-yet-unspecified new Swap extension (which I may rename as Present , as in please present this pixmap in this window ). That s because they aren t related to direct rendering, but rather to tracking window sizes for off-screen rendering, either direct, indirect or even with the CPU to memory. DRI3 and Fences Right now, I m not synchronizing the direct rendering with the CopyArea call; that means the X server will end up with essentially random contents as the application may be mid-way through the next frame before it processes the CopyArea. A simple XSync call would suffice to fix that, but I want a more efficient way of doing this. With the current Linux DRI kernel APIs, it is sufficient to serialize calls that post rendering requests to the kernel to ensure that the rendering requests are themselves serialized. So, all I need to do is have the application wait until the X server has sent the CopyArea request down to the kernel. I could do that by having the X server send me an X event, but I think there s a better way that will extend to systems that don t offer the kernel serialization guarantee. James Jones and Aaron Plattner put together a proposal to add Fences to the X Sync extension. In the X world, those offer a method to serialize rendering between two X applications, but of course the real goal is to expose those fences to GL applications through the various GL sync extensions (including GLARBsync and GLNVfence). With the current Linux DRI implementation, I think it would be pretty easy to implement these fences using pthread semaphores in a block of memory shared between the server and application. That would be DRI-specific; other direct rendering interfaces would use alternate means to share the fences between X server and application. Swap/Present The Second Extension By simply using CopyArea for my application presentation step, I think I ve neatly split this problem into manageable pieces. Once I ve got the DRI3 piece working, I ll move on to fixing the presentation issue. By making that depend solely on existing core Pixmap objects as the source of data to present, I can develop that without any reference to DRI. This will make the extension useful to existing X applications that currently have only CopyArea for this operation. Presentation of application contents occurs in two phases; the first is to identify which objects are involved in the presentation. The second is to perform the presentation operation, either using CopyArea, or by swapping pages or the entire frame buffer. For offscreen objects, these can occur at the same time. For onscreen, the presentation will likely be synchronized with the scanout engine. The second form will mean that the Fences that mark when the presentation has occurred will need to signaled only once the operation completes. A CopyArea operation means that the source pixmap is ready immediately after the Copy has completed. Doing the presentation by using the source pixmap as the new front buffer means that the source pixmap doesn t become ready until after the next swap completes. What I don t know now is whether we ll need to report up-front whether the presentation will involve a copy or a swap. At this point, I don t think so the application will need two back buffers in all cases to avoid blocking between the presentation request and the presentation execution. Yes, it could use a fence for this, but that still sticks a bubble in the 3D hardware where it s blocked waiting for vblank instead of starting on the next frame immediately. Plan of Attack Right now, I m working on finishing up the DRI3 piece:

• Replace the DRI2 buffer allocation kludge with actual local buffer allocation, mapping them into pixmaps using FD passing.
• Replace the DRI2 authentication scheme with having the X server open the DRI object, preparing it for rendering and passing it back to the application.
• Working on the XCB pieces to get the split event-queue stuff landed upstream.
• Implementing the Fencing stuff to correctly serialize access to the pixmap.

The first three seem fairly straight forward. The fencing stuff will involve working with James and Aaron to integrate their XSync changes into the server. After that, I ll start working on the presentation piece. Foremost there is figuring out the right name for this new extension; I started with the name Swap as that s the GL call it implements. However, Swap is quite misleading as to the actual functionality; a name more like Present might provide a better indication of what it actually does. Of course, Present is both a verb and a noun, with very different connotations. Suggestions on this most complicated part of the project are welcome!

Russ Allbery wrote an informative post about how to determine which charities are worth donating to [1]. He has a link to another article about the charities to which he donates and concentrates on ways of analysing the effectiveness of charities. So someone who has different ideas about which types of charity are worthy of donation could still learn a lot from his post. Adam Green wrote an interesting article for The New Yorker about Apollo Robbins who is one of the world s best pick-pockets [2]. Apollo picks pockets as a magician to entertain people and always returns what he steals. Now he is working with neuroscientists who are devising experiments to determine why his tricks work. Rick Falkvinge wrote an insightful article describing the way that the copyright monopoly is in direct opposition to the freedom to make contracts [3]. It s a good rebuttal of a common argument in favor of copyright law. Seth Godin gave an interesting TED talk about the problems with the education system, how and why it teaches conformity and little else [4]. One of his suggestions for improvement is to have students spend their evenings watching lectures by experts and class time asking questions. He also says that everything should be open book and that there is no value in memorising anything it s a bit of an overstatement but it s essentially correct. Cory Doctorow wrote an interesting article for The Guardian about positive externalities and copyright law [5]. I think that he didn t choose the best way of framing this issue, but he makes some very interesting points anyway. Andrew Norton wrote an interesting article about how to reduce corruption in the police force and other government agencies [6]. A large part of this is based on making them subject to the same laws as everyone else, which seems to be a radical idea. Valerie Aurora wrote an insightful blog post about suicide [7]. Emily Oster gave an interesting TED talk about the factors that determine the spread of AIDS in Africa [8]. It s quite different to what you probably expect.

• [1] http://www.eyrie.org/~eagle/journal/2012-12/006.html
• [2] http://www.newyorker.com/reporting/2013/01/07/130107fa_fact_green
• [3] http://tinyurl.com/apovrka
• [4] http://www.youtube.com/watch?v=sXpbONjV1Jc
• [5] http://tinyurl.com/a7rutft
• [6] http://tinyurl.com/a227fvg
• [7] http://tinyurl.com/a5zo6tn
• [8] http://tinyurl.com/3f7283s

On Tuesday, there was a memorial for Aaron Swartz held at the MIT Media Lab. Unfortunately, I am traveling this week and was unable to attend. As I wrote recently, I was close to Aaron. I am also, more obviously, close to MIT and the lab. It was important to me to participate in the memorial and I found a way to give a short talk with a video. I think the lab plans to post a recording of the whole event but I have put the video of my own remarks below (and online in WebM). If you prefer, you can also read the text of the remarks. <iframe allowfullscreen="" frameborder="0" height="315" src="http://www.youtube.com/embed/uYGoAJdE8jg" width="420"></iframe>

Aaron on Software wrote an interesting series of blog posts about psychology and personal development collectively Titled Raw Nerve , here s a link to part 2 [1]. The best sections IMHO are 2, 3, and 7. The Atlantic has an insightful article by Thomas E. Ricks about the failures in leadership in the US military that made the problems in Afghanistan and Iraq a lot worse than they needed to be [2] Kent Larson gave an interesting TED talk about how to fit more people in cities [3]. He covers issues of power use, transport, space use, and sharing. I particularly liked the apartments that transform and the design for autonomous vehicles that make eye contact with pedestrians. Andrew McAfee gave an interesting TED talk titled Are Droids Taking Our Jobs [4]. I don t think he adequately supported his conclusion that computers and robots are making things better for everyone (he also presented evidence that things are getting worse for many people), but it was an interesting talk anyway. I Psychopath is an interesting documentary about Sam Vaknin who is the world s most famous narcissist [5]. The entire documentary is available from Youtube and it s really worth watching. The movie Toy Story has been recreated in live action by a couple of teenagers [6]. That s a huge amount of work. Rory Stewart gave an interesting TED talk about how to rebuild democracy [7]. I think that his arguments against using the consequences to argue for democracy and freedom (he suggests not using the torture doesn t work and women s equality doubles the workforce arguments) are weak, but he made interesting points all through his talk. Ernesto Sirolli gave an interesting TED talk about aid work and development work which had a theme of Want to help someone? Shut up and listen! [8]. That made me think of Mary Gardiner s much quoted line from the comments section of her Wikimania talk which was also shut up and listen . Waterloo Labs has some really good engineering Youtube videos [9]. The real life Mario Kart game has just gone viral but there are lots of other good things like the iPhone controlled car and eye controlled Mario Brothers. Robin Chase of Zipcar gave an interesting TED talk about various car sharing systems (Zipcar among others), congestion taxes, the environmental damage that s caused by cars, mesh networks, and other things [10]. She has a vision of a future where most cars are shared and act as nodes in a giant mesh network. Madeleine Albright gave an interesting TED talk about being a female diplomat [11]. She s an amazing speaker. Ron Englash gave an interesting TED talk about the traditional African use of fractals [12]. Among the many interesting anecdotes concerning his research in Africa he was initiated as a priest after explaining Georg Cantor s set theories. Racialicious has an insightful article about the low expectations that members of marginalised groups have of members of the privileged groups [13]. Rick Falkvinge has a radical proposal for reforming copyrights with a declared value system [14]. I don t think that this will ever get legislative support, but if it did I think it would work well for books and songs. I think that some thought should be given to how this would work for Blogs and other sources of periodical content. Obviously filing for every blog post would be an unreasonable burden. Maybe aggregating a year of posts into one copyright assignment block would work. Scott Fraser gave an interesting TED talk about the problem with eyewitness testimony [15]. He gave a real-world example of what had to be done to get an innocent man acquitted, it s quite amazing. Sarah Kendzior wrote an interesting article for al Jazeera about the common practice in American universities to pay Adjunct Professors wages that are below the poverty line [16]. That s just crazy, when students pay record tuition fees there s more than enough money to pay academics decent wages, where does all the money go to anyway?

• [1] http://www.aaronsw.com/weblog/dweck
• [2] http://tinyurl.com/bxtvl2f
• [3] http://tinyurl.com/8nz8zjr
• [4] http://tinyurl.com/cpu7ahx
• [5] http://topdocumentaryfilms.com/i-psychopath/
• [6] http://tinyurl.com/aejpwkr
• [7] http://www.ted.com/talks/rory_stewart_how_to_rebuild_democracy.html
• [8] http://tinyurl.com/cbv7lvc
• [9] http://www.youtube.com/user/WaterlooLabs
• [10] http://www.ted.com/talks/robin_chase_on_zipcar_and_her_next_big_idea.html
• [11] http://tinyurl.com/48dfthu
• [12] http://www.ted.com/talks/ron_eglash_on_african_fractals.html
• [13] http://www.racialicious.com/2012/12/10/privilege-and-low-expectations/
• [14] http://falkvinge.net/2012/12/10/declared-value-system/
• [15] http://tinyurl.com/d8v9zax
• [16] http://tinyurl.com/8bdmn3p

It has been so very long since I have left a trace here. I guess moving to two new countries (Canada and Quebec), starting a new job, working on Anonymous, and finishing my first book was a bit much. I miss this space, not so much because what I write here is any good. But it a handy way for me to keep track of time and what I do and even think. My life feels like a blur at times and hopefully here I can see its rhythms and changes a little more clearly if I occasionally jot things down here. So I thought it would nice to start with something that I found surprising: famed information designer, Edward Tufte, a professor emeritus at Yale was a phone phreak (and there is a stellar new book on the topic by former phreak Phil Lapsley. He spoke about his technological exploration during a sad event, a memorial service in NYC which I attended for the hacker and activist Aaron Swartz. I had my wonderful RA transcribe the speech, so here it is [we may not have the right spelling for some of the individuals so please let us know of any mistakes]:

Edward Tufte s Speech From Aaron Swartz s Memorial
Speech starts 41:00 [video cuts out in beginning]
We would then meet over the years for a long talk every now and then, and my responsibility was to provide him with a reading list, a reading list for life and then about two years ago Quinn had Aaron come to Connecticut and he told me about the four and a half million downloads of scholarly articles and my first question is, Why isn t MIT celebrating this? .
[Video cuts out again]
Obviously helpful in my career there, he then became president of the Mellon foundation, he then retired from the Mellon foundation, but he was asked by the Mellon foundation to handle the problem of JSTOR and Aaron. So I wrote Bill Bullen(sp?) an email about it, I said first that Aaron was a treasure and then I told a personal story about how I had done some illegal hacking and been caught at it and what happened. In 1962, my housemate and I invented the first blue box, that s a device that allows for free, undetectable, unbillable long distance telephone calls. And we got this up and played around with it and the end of our research came when we concluded what was the longest long distance call ever made, which was from Palo Alto to New York time-of-day via Hawaii, well during our experimentation, AT&T, on the second day it turned out, had tapped our phone and uh but it wasn t until about 6 months later when I got a call from the gentleman, AJ Dodge, senior security person at AT&T and I said, I know what you re calling about. and so we met and he said You what you are doing is a crime that would , you know all that. But I knew it wasn t serious because he actually cared about the kind of engineering stuff and complained that the tone signals we were generating were not the standard because they record them and play them back in the network to see what numbers they we were that you were trying to reach, but they couldn t break though the noise of our signal. The upshot of it was that uh oh and he asked why we went off the air after about 3 months, because this was to make long distance telephone calls for free and I said this was because we regarded it as an engineering problem and we made the longest long distance call and so that was it. So the deal was, as I explained in my email to Bill Bullen, that we wouldn t try to sell this and we were told, I was told that crime significance would pay a great deal for this, we wouldn t do any more of it and that we would turn our equipment over to AT&T, and so they got a complete vacuum tube isolator kit for making long distance phone calls. But I was grateful for AJ Dodge and I must say, AT&T that they decided not to wreck my life. And so I told Bill Bullen that he had a great opportunity here, to not wreck somebody s life, course he thankfully did the right thing.
Aaron s unique quality was that he was marvelously and vigorously different. There is a scarcity of that. Perhaps we can be all a little more different too.
Thank you very much.

On praising people, and on success This morning I was pointing out to friends how excellent is mako's post on Aaron Swartz, and I thought it'd be nice if we didn't have to wait for people to die before telling the world how awesome and inspirational they are. Then Russ posted an article about work, success and motivation and I went to tell my friends how awesome and inspirational he is. I, too, see myself as somehow successful, and I, too, don't identify in the usual stereotype of success. I don't want to stop being a craftsman to become a manager, I don't get a high from having power over other people, I don't define my value in terms of my profits. At a glance, people don't see me as successful, until they get to know me better. They they realise that I'm not at all unhappy about my life. I have a job that I like, I write Free Software and it gets used and appreciated, my colleagues are friends, who respect me and my opinion, and I respect them and theirs. I can work from home. In fact, I can work from everywhere as long as I have my laptop with me. I can sustain a long distance relationship because I can work from the house of my partner when I'm visiting. Two days ago I worked from the bar of a farm on top of a hill, because I was on the road, it was close by, and what the hell, it's a wonderful place to be. To me success means that I can care about the quality of my life, that I have the luxury of caring about little things that make my day, of trying to make good ideas sustainable, of working a bit more when I'm on fire, and of working a bit less when there's something wonderful in the world to see, or someone interesting in the world to meet. Russ, the way I read your article, you are questioning what "success" means, and you are spot on. People should be able to define "success" as whatever works for them and pursue it freely. Only then success becomes something that is worth praising when it is achieved. Only then it becomes inspirational. I like how you managed to put into words something that has been for a long time in some corner of my mind and I hadn't yet managed or bothered to bring into the spotlight. You have the insight and the confidence of seeing something in an insightful but non-mainstream way, and say "you know what? That actually makes sense." Sometimes I read your post, nod a lot and realise how important something actually is, how that is actually such an important part of myself. And now that you took it out for me to see it, I can appreciate how valuable it is, and make sure I don't accidentally lose it. Thanks! That's another one I owe you. It's just the kind of thing I shouldn't wait before letting you know.

I moved to Boston in 2005 at the same time that Aaron Swartz did and we were introduced by a mutual friend. Aaron was one of my first friends in Boston and we became close. When Aaron moved to San Francisco, I moved into his apartment in Somerville where he kept a room for a year or so. Mika and I still live there. His old posters remain on our walls and his old books remain on our shelves. Aaron s brothers Ben and Noah both lived with us and remain close friends. I have spent hours (days?) reading and thinking about Aaron over the last two weeks. It has been disorienting but beautiful to read the descriptions of, and commentaries on, Aaron s life. Although I suspect I may never feel ready, there are several things I want to say about Aaron s death, about Aaron s work, and about what Aaron means to me. 1. Aaron s Death The reaction to Aaron s death has been overwhelming and inspirational. At some point in the near future I plan to join some of the important campaigns already being waged in his name. There are many attempts to understand why Aaron died and many attempts to prevent it from happening to others in the future. Unfortunately, I am familiar with the process of soul-searching and second-guessing that happens when a friend commits suicide. I m sure that every one of his friends has asked themselves, as I have, What could I have done differently? I don t know the answer, but I do know this: Aaron was facing the real risk of losing half his life to prison. But even if one believed that he was only facing the likely loss of ten percent or even one percent of his life I wish that we all, and I wish that I in particular, had reacted with the passion, time, anger, activity, and volume proportional to how we have reacted in the last two weeks when he lost the whole thing. 2. Aaron s Work Of course, Aaron and I worked on related projects and I followed his work. And despite all the incredible things that have been said about Aaron, I feel that Aaron s work was more focused, more ambitious, more transformative, more innovative, and more reckless (in a positive sense) than the outpour online suggests. Although discussion of Aaron has focused his successes, achievements, and victories, the work that inspired me most was not the projects that were most popular or successful. Much of Aaron s work was deeply, and as it turned out overly, ambitious. His best projects were self-conscious attempts to transform knowledge production, organization, and dissemination. Although he moved from project to project, his work was consistently focused on bringing semantic-web concepts and technologies to peer production, to the movement for free culture, and to progressive political activism and on the meta-politics necessary to remove barriers to this work. For example, Aaron created an online collaborative encyclopedia project called the TheInfoNetwork (TIN) several years before Wikipedia was started. I talked to Aaron at length about that project for a research project I am working on. Aaron s work was years ahead of its time; in 2000, TIN embraced more of the Wikimedia Foundation s current goals and principles than Wikipedia did when it was launched. While Wikipedia sought to create a free reference work online, Aaron s effort sought to find out what a reference work online could look like. It turned out to be too ambitious, perhaps, but it taught many, including myself, an enormous amount in that process. When I met Aaron, he was in the process starting a company, Infogami, that was trying to chase many of TIN s goals. Infogami was conceived of as a wiki aware of the structure of data. The model was both simple and profound. Years later, Wikimedia Deutschland s WikiData project is beginning to bring some of these ideas to the mainstream. Infogami merged with Reddit as equal halves of a company with a shared technological foundation based on some of Aaron s other work. But when Reddit took off, Infogami was rarely mentioned, even by Aaron. I think that is too bad. Reddit got traction because it made the most popular stuff more visible; Reddit is popular, fundamentally, because popular things are popular. But popular is not necessary positive. For that reason, Reddit never struck me as either surprising or transformative. But what started as Aaron s half the company, on the other hand, aimed to create a powerful form of democratized information production and dissemination. And although Infogami didn t take off, the ideas and code behind the project found life at the heart of Open Library and will continue to influence and inspire countless other projects. I believe that Infogami s lessons and legacy will undergird a generation of transformative peer production technologies in a way that the Reddit website important as it is will not. 3. What Aaron Means to Me A lot of what has been written about Aaron speaks to his intelligence, his curiosity, his generosity, his ethics and his drive. Although I recognize all these qualities in the Aaron I knew, I ve felt alienated by how abstract some of the discussion of Aaron has been my memories are of particularities.

I remember the time Aaron was hospitalized and I spent two hours on the phone going through my bookshelves arguing with him about the virtues of the books in my library as we tried to decide which books I would bring him. I remember Aaron confronting Peter Singer intellectual founder of the modern animal rights movement at the Boston Vegetarian Food Festival to ask if humans had a moral obligation to stop animals from killing each other. I lurked behind, embarrassed about the question but curious to hear the answer. (Singer sighed and said yes sort of and complemented Aaron on the enormous Marxist commentary he was carrying.) I remember 1-800-INTERNET.com. I remember talking with Aaron about whether being wealthy could be ethical. I argued it could not but Aaron argued uncharacteristically I thought that it could. Aaron told Mika she should slap him if he ever became wealthy. The very next day, it was announced that his company had been acquired and that Aaron was a millionaire. I remember the standing bets I had with Aaron and how he would email me every time news reports favored his claims (but never when they did not). And I remember that I won t hear from him again.

Aaron was a friend and inspiration. I miss him deeply and I am very sad.

Daniel Kahn Gillmor has a very good blog post. You should read it. It includes a thoughtful rebuttal to some of my earlier thoughts about activism. I think I'm developing a richer understanding of where I see boundaries here, but after my last post, I also realized that by focusing on the specific details of what should have been a minor alleged crime, I'm derailing. Swartz did so much else. I made a note to come back to the more theoretical discussion in six months; now isn't the time. Now is the time to celebrate open content and all of the things Swartz achieved. (But thank you very much to the multiple people who have pointed out flaws in my reasoning and attempted approach.) Hopefully, it's also an opportunity to keep the pressure on for a saner and less abusive judicial system that doesn't threaten people with ridiculous and disproportional punishment in order to terrify them into unwarranted plea bargains. The petition I mentioned has reached nearly 40,000 signatures and passed the threshold (at the time it was posted) for forcing a White House response. Probably more importantly, it also seems to be creating the feedback cycle that I was hoping to see: the popularity of the petition is causing this story to stay in the news cycle and continue to be written about, which in turn drives more signatures to the petition. I'm not particularly hopeful that the Obama administration cares about the vast and deep problems with our criminal justice system, but I'm somewhat more hopeful that they, like most politicians, hate news cycles that they don't control. The longer this goes on, the stronger the incentive to find some way to make it go away, which could lead to real disciplinary action. A key committee in the US House of Representatives is starting a formal investigation. One of my local representatives has proposed modifying the US federal law on computer fraud and abuse to remove violations of terms of service from the definition of the crime. (I don't have much hope that this will pass when proposed by the minority party in a fairly hostile House, but the mere act of proposing it keeps the news focus on.) Glenn Greenwald has a (typically long-winded) round-up of news in the Guardian. Note that both Greenwald and Declan McCullagh link directly to the petition in articles in mainstream news outlets. One thing that slacktivism can do is perpetuate a news cycle until it gets more uncomfortable for people in power. It's still nowhere near as effective as the types of activism that Swartz was so good at, but in this specific case I think one gets a reasonable return on one's five-minute investment of effort. I'm going to stop talking about this now, since other people are a lot better at this sort of post than I am. But one last link: the medical community has a related open content problem, and theirs is also killing people. Possibly people you know. If all of this has inspired you, as it has me, to care even more about open content, be watching the push for open access to clinical trial data. More background is in Ben Goldacre's TED talk.

I just returned home from Aaron Swartz s funeral in Chicago. Aaron was a good friend. The home I ve returned to is an apartment that was Aaron s before it was mine, that I have lived in with Aaron during several stints, and that I still share with many of his old books and posters. Although, I ve spent what feels like most of the last five days reading things that people have written about Aaron, I m still processing and digesting myself. Right now, I m very sad and at a loss for words. While I reflect, I wanted to share this video recently put online by Finne Boonen. The video was made in 2006 at a Web 1.0 Elevator Pitch Competition held at Wikimania 2006 about a year after that both Aaron and I moved to Cambridge and met. The goal of the contest was to pitch Web 1.0 DotCom business ideas to a team of real Web 1.0 investors like it was still 1999. Aaron and I formed a team along with SJ Klein (who I traveled to the funeral with this week), and Wikimania general counsel and interim executive director Brad Patrick. The video shows how as Danny O Brien has reminded us Aaron was funny. He came up with many our teams best lines in addition to checking our Web 1.0 boxes for tech guru and Stanford dropout. Our pitch for 1-800-INTERNET.COM is in the video below. The transcript was done by Phoebe Ayers in Facebook and the video is also available in WebM. <iframe allowfullscreen="" frameborder="0" height="315" src="http://www.youtube.com/embed/NzFjWY6Fd_g" width="420"></iframe>

SJ: You know, Mako and I had some pretty good ideas for improving connectivity to the internet, and we think we can reach 90% of the world s population. So think about this. You re sitting in a Starbucks, and you need to connect to the internet. But you can t, because there s no internet. But what is there, near every Starbucks? There s a payphone! You pick up the payphone, and you call . 1-800-INTERNET. You can connect to our bank of researchers on our fast T1 connections and get any information you need! So, we don t actually have 1-800-INTERNET yet, we have 1-800-225-3224, so the first thing we need to do is buy the number. So here s Mako, who is our web designer from UC Santa Cruz and Bradford, our financial guru, and Aaron, who s handling all of our technical implementation. But Mako, you should explain the earballs. Mako: So, so, so yeah, so most people on the Internet are going for the eyeballs, but they ve just left all of these earballs. So I have some experience in web design, and it s true that this isn t really a website, but we still need good web design. So, so, I ve actually got a really experienced team, we can go into later, and we have some really great earcons not icons, but earcons.. And it s going to be all together, not apart like some of the websites. It s going to be together. Brad: so how does this work technically? Aaron: Well, I mean, so I only spent one year at Stanford but that s Ok, because there are new developmental technologies, we re going to throw away all that old stuff, we re going to use really reliable and efficient well-designed code that everyone can clearly understand, and write the whole thing in Perl. I know this is a risk, but I am confident that Perl is going to destroy those old C websites. No one will write websites in C anymore once we do this, it s going to be so much faster, and so dynamic, everythings going to be like, on top of everything. It s going to be great. Bradford: So here s the business model. It s really really simple, and it s a really really great idea. It s all about the licensing. Because what we re going to have are these underlying audio ads, While you re on the phone you re going to hear this subliminal advertising message. And the way it works is really really cool, because it s really really low volume, it s high impact! And it s even better, because we license it, and the way it works is when a caller calls 1-800-Internet, they re hearing the ad, but so is the representative, so we get to bill em twice! So that s it: All: 1-800-INTERNET.COM

We did not win and I still believe that we were robbed.

In order to move my blog to a free-as-in-freedom platform and support the great work that Joey (of git-annex fame) and Lars (of GTD for hackers fame) have put into their service, I decided to convert my Blogger blog to Ikiwiki and host it on Branchable. While the Ikiwiki tips page points to some old instructions, they weren't particularly useful to me. Here are the steps I followed.

Exporting posts and comments from Blogger Thanks to Google letting people export their own data from their services, I was able to get a full dump (posts, comments and metadata) of my blog in Atom format. To do this, go into "Settings Other" then look under "Blog tools" for the "Export blog" link.

Converting HTML posts to Markdown Converting posts from HTML to Markdown involved a few steps:

1. Converting the post content using a small conversion library to which I added a few hacks.
2. Creating the file hierarchy that ikiwiki requires.
3. Downloading images from Blogger and fixing their paths in the article text.
4. Extracting comments and linking them to the right posts.

The Python script I wrote to do all of the above will hopefully be a good starting point for anybody wanting to migrate to Ikiwiki.

Maintaining old URLs In order to make sure I wouldn't break any existing links pointing to my blog on Blogger, I got the above Python script to output a list of Apache redirect rules and then found out that I could simply email these rules to Joey and Lars to get them added to my blog. My rules look like this:

# Tagged feeds
Redirect permanent /feeds/posts/default/-/debian http://feeding.cloud.geek.nz/tags/debian/index.rss
Redirect permanent /search/label/debian http://feeding.cloud.geek.nz/tags/debian
# Main feed (needs to come after the tagged feeds)
Redirect permanent /feeds/posts/default http://feeding.cloud.geek.nz/index.rss
# Articles
Redirect permanent /2012/12/keeping-gmail-in-separate-browser.html http://feeding.cloud.geek.nz/posts/keeping-gmail-in-separate-browser/
Redirect permanent /2012/11/prefetching-resources-to-prime-browser.html http://feeding.cloud.geek.nz/posts/prefetching-resources-to-prime-browser/

Collecting analytics Since I am no longer using Google Analytics on my blog, I decided to take advantage of the access log download feature that Joey recently added to Branchable. Every night, I download my blog's access log and then process it using awstats. Here is the cron job I use:

#!/bin/bash
BASEDIR=/home/francois/documents/branchable-logs
LOGDIR=/var/log/feedingthecloud
# Download the current access log
LANG=C LC_PAPER= ssh -oIdentityFile=$BASEDIR/branchable-logbot b-feedingthecloud@feedingthecloud.branchable.com logdump > $LOGDIR/access.log

It uses a separate SSH key I added through the Branchable control panel and outputs to a file that gets overwritten every day. Next, I installed the awstats Debian package, and configured it like this:

$ cat /etc/awstats/awstats.conf.local
SiteDomain=feedingthecloud.branchable.com
LogType=W
LogFormat=1
LogFile="/var/log/feedingthecloud/access.log"

Even if you're not interested in analytics, I recommend you keep an eye on the 404 errors for a little while after the move. This has helped me catch a critical redirection I had forgotten.

Limiting Planet feeds One of the most common things that happen right after someone migrates to a new blogging platform is the flooding of any aggregator that subscribes to their blog. The usual cause being the change in post identifiers. Unsurprisingly, Ikiwiki already had a few ways to avoid this problem. I chose to simply modify each tagged feed and limit them to the posts added after the move to Branchable.

Switching DNS Having always hosted my blog on a domain I own, all I needed to do to move over to the new platform without an outage was to change my CNAME to point to feedingthecloud.branchable.com. I've kept the Blogger blog alive and listening on feeding.cloud.geek.nz to ensure that clients using a broken DNS resolver (which caches records for longer than requested via the record's TTL) continue to see the old posts.

I was upset to learn about Aaron Swartz's death last week. I continue to be upset about his loss, and about our loss. He didn't just show promise of great things to come in the future -- he had already done more work for the public good than many of us will ever do. I'd only met him IRL a couple times, but (like many others) i had encountered him on the 'net in many places. He was a good person, someone i didn't need to always agree with to respect. I read Russ Allbery's posts about Aaron and "slacktivism" with much appreciation. I had been ambivalent about signing the whitehouse.gov petition asking for the removal of the prosecutor for overreach, because I generally distrust the effectiveness of online petitions (and offline petitions, for that matter). But Russ's analysis convinced me to go ahead and sign it. The petition is concrete, clear (despite wanting a grammatical proofread), and actionable. For people willing to go beyond petition signing to civil disobedience, The Aaron Swartz Memorial JSTOR Liberator is an option. It makes it straightforward to potentially violate the onerous JSTOR terms of service by re-publishing a public-domain article from JSTOR to archive.org, where it will be accessible to anyone directly. As someone who builds and maintains information/communications infrastructure, i have very mixed feelings about most online civil disobedience, since it often takes the form of a Distributed Denial of Service (DDoS) attack of some sort. DDoS attacks of public services are notoriously difficult to defend against without having huge resources to throw at the problem, so encouraging participation in a DDoS often feels a little bit like handing out cans of gasoline when you know that everyone is living in a house of straw. However, the JSTOR Liberator is not a DDoS at all -- it's simply a facilitation of people breaking the JSTOR Terms of Service (ToS), some of the same terms that Aaron was facing charges for violating. So it is a well-targeted way to demonstrate that the prosecutions were overreaching. I wanted to take issue with one of Russ' statements, though. In his second post about the situation, Russ wrote:

Social activism and political disobedience are important and often valuable things, but performing your social activism using other people's stuff is just rude. I think it can be a forgivable rudeness; people can get caught up in the moment and not realize what they're doing. But it's still rude, and it's still not the way to go about civil disobedience.

While i generally agree with Russ' thoughtful consideration of consent, I have to take issue with this elevation of some sort of hyper-extended property right over the moral agency that drives civil disobedience. To use someone else's property for the sake of a just cause without damaging the property or depriving the owner of its use is not "forgivable rudeness" -- it's forgivable, laudable even, because it is just. And the person using the property doesn't need to be "caught up in the moment and not realize what they're doing" for it to be acceptable. Civil disobedience often involves putting some level of inconvenience or discomfort on other people, including innocent people. It might be the friends and family of the activist who have to deal with the jail time; it might be the drivers stuck in a traffic jam caused by a demonstration; it might be the people forced to shop elsewhere because the store's doors are barricaded by protestors. All of these people could be troubled by the civil disobedience more than MIT's network users and admins were troubled by Aaron's protest, and that doesn't make the protests described worse or "not the way to go about civil disobedience." The trouble highlights a more significant injustice, and in its troubling way does what it can to help right it. Aaron was a troublemaker, and a good one. He will be missed. Tags: aaronsw

I got some feedback that the analogy I used in my last post was a bit confusing, and indeed I blew the phrasing of the analogy (also now corrected). So let me try this again, since I think there's a subtlety here that may be missed. I should note for the record that my understanding of what Swartz did that started the process is apparently somewhat based on the description from the prosecution, so it may not be the complete or accurate facts. Since there will now be no trial, we may never find out what the defense was, and whether those facts would be challenged. So it may be best to think of this as a hypothetical. We never established, or will establish, in court exactly what happened. Swartz was, generally speaking, charged with two things that I consider quite distinct, at least from an ethical perspective. Most of the focus is on the copyright part: downloading JSTOR articles with an intent (never acted upon) to distribute them to the world. There are a bunch of reasons why this may or may not be justified, which are tied into the origin of those articles (many of them were publicly funded) and the legitimacy of copyright licensing agreements. I think there is significant room to hold a variety of opinions on this, although I don't believe that "crime worth 35 years in prison" is even remotely close to justifiable under any interpretation. However, there is a second part of what he was charged with, and that's primarily what I was commenting on, not the copyright part. He allegedly hooked up a laptop in an unlocked MIT wiring closet without permission and then used MIT's network and JSTOR license to download information. This is, to me, a subtlely but entirely distinct act from the question of whether taking JSTOR's data was ethically or legally wrong. Whether or not one believes that JSTOR's copyrights are not legitimate, it's still not okay to use someone else's network and license or to trespass in their wiring closets without permission. (I work in central IT for a university, so this strikes closer to home.) And this is where the analogy came in, which I flubbed. I had originally said that this was akin to "traipsing into someone's barn or backyard shed without their permission and taking some of their tools because you want to use them." The word "taking" was wrong; it falsely implies that you weren't going to return the tools. I had been thinking "taking and then returning" when I wrote that, and the important second part didn't make it into the post. So let's try this again: Swartz's actions at MIT, as I understand them (and many things could change this, such as a revelation that he had MIT's prior permission), are akin to going onto someone's property and into their barn without permission, borrowing their hedge trimmer for a while because you want to use it, and returning the hedge trimmer without any damage when you're done (and without them noticing it was gone). I'm quite fond of this analogy, since I think it clearly establishes two things:

1. Most people are going to be unhappy about this happening and will intuitively feel like it should probably be illegal. Not everyone; there are folks who don't believe in personal property, or at least wouldn't extend it to tools in a barn. But most people will feel that someone should ask first before they come borrow your tools. Even if they don't damage them, even if they return them before you notice they're gone, you might have wanted to use the tool at the same time or they might have damaged them without intending to, and they should just ask first. It's common politeness, and depending on the circumstances, someone who doesn't ask and is covert about borrowing tools might be worth calling the police over.
2. There is absolutely no way in any reasonable moral system that doing this should result in 35 years in prison. Or even 10, or even 1. Yes, most people would consider this a crime, but most people would consider it a minor crime. It's the sort of thing where you might have to impose some consequences just to make sure the message of "knock it off" is delivered firmly, but someone doing this is rude and inconsiderate, not evil.

The JSTOR copyright stuff is more complex to analyze and is more politically divisive, but for me the key points are (a) Swartz never released the data, and (b) JSTOR declined to press charges. To me, that means the deeper copyright questions, which are quite interesting, were never actually reached in this particular case. The crime that did apparently happen was the trespass at MIT, for which I think the above analogy is the right way to think about it. The point I do want people to take away from this is that one should not overlook the trespass at MIT even if one wants to celebrate the undermining of the copyright regime and doesn't believe JSTOR's data should be considered their private property. Social activism and political disobedience are important and often valuable things, but performing your social activism using other people's stuff is just rude. I think it can be a forgivable rudeness; people can get caught up in the moment and not realize what they're doing. But it's still rude, and it's still not the way to go about civil disobedience. For both ethical and tactical reasons, involving bystanders in your act of social activism without their consent is a bad approach. ETA: The problem, of course, with discussing all of this is that while it's relevant and possibly even somewhat important in the broader sense of how our community acts going forward, it also doesn't capture the fact that this was only one incident in a remarkable life. One of the worst problems with the abusive prosecution of Swartz is that it blew this incident completely out of proportion. A moment of arguable judgement should not dominate one's life or cast a shadow over all of one's other accomplishments; the prosecution tried to make it do just that. That's part of what I'm arguing, but ironically that partly feeds into the lack of proportionality in the discussion. I've gotten kind of far afield here, so let me go back and say explicitly: Swartz was a remarkable person who did much to admire and respect, and the world is a worse place without him.

Normally, I'm not that much of a fan of the slacktivist trend of signing pre-canned petitions, signing things that political organizations send in the mail, and so forth. It takes very little effort and therefore carries very little weight. A lot of those efforts are more exercises in helping the signers feel better about themselves. But there is the occasional exception. Most of you probably already know about the death of Aaron Swartz. For those who aren't familiar, see Larry Lessig's article. Swartz suffered from depression (and, seriously, fuck depression it's an awful, horrible disease), but there's little doubt that the ongoing federal prosecution using the full weight of the US district attorney's office to hound him for what amounts to political trespassing was part of what led to his suicide. As it happens, I personally believe that Swartz committed a crime, and probably should have paid some consequence for it (on the order of a fine or some community service). Hooking your devices up to someone else's network without their permission and messing around in their wiring closets (locked or not) is, for me, akin to traipsing into someone's barn or backyard shed without their permission and using some of their tools because you want to use them. And whether or not one likes the current copyright regime (and I don't like it at all), MIT is still in the awkward position of having to work with it. Abusing their license for your political goals is effectively recruiting them into your activism without their permission, and as I've mentioned before, I have a mild obsession with consent. It was a crime. However, it was a minor crime, and that's where this whole situation went completely off the rails. One aspect of a justice system is fairness: uniform application of the laws to everyone. However, another aspect of a justice system is proportionality: punishments that fit the crime. Without ever having been convicted, Swartz was already punished completely out of proportion to what he actually did, both financially and emotionally, by prosecution by the US attorney that went far, far beyond zealous into actively abusive. This despite the fact that the owner of the academic papers that he was downloading as an act of open access activism stated they did not want the case to proceed and asked the US government to drop the charges. I don't think what Swartz did was right, or legal. However, the correct reaction was "look, involuntarily recruiting MIT as an accomplice to your act of civil disobedience is not okay don't ever do that again." Not "you are evil and should be locked up in prison for 35 years." And I'm completely fed up with the disproportionality of our justice system and the practice of ridiculous over-charging of crimes in an attempt to terrify people into bad plea bargains. Which brings me to the slacktivism. This is exactly the sort of situation where popular opinion matters. US attorneys who have lost the faith and support of the population they serve won't keep their jobs. And no one in government particularly wants this case to be splashed across the front pages, or to have to answer questions about the appropriateness or proportionality of the prosecution while people are mourning a dead young man. If we make it clear enough to the Obama administration that people are watching, that this matters, and that we're angry about it, not only is it quite likely there will be consequences for this prosecutor, but it may serve as a deterrant for other prosecutors in the future. There is a petition on WhiteHouse.Gov to remove the district attorney for prosecutorial overreach, and for once taking five minutes to create an account and clicking on a petition may be both useful and appropriate. Taking disciplinary action here is, unlike with a lot of petitions, something that the Obama administration can actually do, directly, without involving the rest of the dysfunctional US government, and without making new law. If you are a US citizen, please consider going to this site and signing it to say that this matters to you and you believe this prosecution was excessive and inappropriate. Please note: you do not have to believe that Swartz died solely or even primarily because of this prosecution to do this. We'll never really know the complex factors behind his death. But this was a burden that he shouldn't have had to deal with. Please also note that you do not have to think he was justified in his behavior to sign this petition. This is not a question of whether what he did should be legal or was ethical. Rather, even assuming it was illegal, it's a question of appropriate punishment and proportionality of response. Whether or not you agree with his political cause, there simply was not enough damage done, to anyone, to warrant this kind of aggressive prosecution. And the direct victims agreed, which was the point at which the district attorney should have scaled way back on their actions or dropped the matter entirely. Not doing so was an abuse of office and position, and that should have consequences. ETA: Corrected "taking" to "using" in the analogy about tools. I was actually thinking "taking and then returning" when I wrote that, but only the first word made it into the post, and ended up creating a confusing parallel with theft that wasn't intended.

Aaron was an inspiration to me personally, politically, and professionally ever since we met (ice cream and word games with a small group in a bank vault at Herrell's in Harvard Square) several years ago. I don't understand how things got to this point, but I know I'm angry along with Lessig. I'm so sorry for all of his family and friends; all the rest of us can do is try to make even a tiny sliver of the difference he did.

I recently had a chance to briefly play with the new Samsung Galaxy Camera [1]. The Galaxy Camera is an Android device with a 4.8 display (the same size as the Samsung Galaxy S3) that has a fairly capable camera (IE nothing like a typical phone camera). It runs Android 4.1 (Jelly Bean) and the camera has 21* zoom with a 16 megapixel sensor. Camera Features It seems that professional photographers are often annoyed when they see someone with a DSLR set in auto mode. It s widely regarded that auto mode is a waste of a good camera, although the better lenses used with DSLRs will usually give a better result than any compact camera even when it s in auto mode. The problem is that photography is quite complex, in an earlier post about digital cameras I summarised some of the technical issues related to cameras and even without any great detail it became a complex issue [2]. The Galaxy Camera has a reasonably friendly GUI for changing camera settings which even includes help on some of the terms, I expect that most people who use it will end up using most of the features which could make it a good training camera for someone who is going to move to a DSLR. A DSLR version of the Galaxy Camera could also be an interesting product. The camera also has modes such as Waterfall and Panorama , hopefully the settings for those would be exposed to the user so they could devise their own similar groups of settings. I ve seen the phone criticised for the lack of physical controls as the expert mode in software is inherently slower than manually turning dials on a DSLR. But it seems obvious to me that anyone who knows how to use the controls manually should be using a DSLR or bridge camera and anyone who doesn t already know how to do such things will be better suited by the software controls. It supports 120fps video at 720*480 resolution (with a file format stating that it s 30fps to give 1/4 speed) which could be useful. I used to have a LG Viewty smart-phone that did 120fps video but the resolution was too low to be useful. 720*480 is enough resolution to see some detail and has the potential for some interesting video, one use that I ve heard of is filming athletes to allow them to analyse their performance in slow motion. It also does 60fps video at 720p (1280*720) resolution. One down-side to the device is that the lens cover doesn t seem particularly sturdy. It s quite OK for a device that will be stored in a camera case but not so good for a device that will be used as a tablet. I didn t get to poke at the lens cover (people don t like it if you mess with their Christmas presents) but it s design is a couple of thin flaps that automatically retract when the camera is enabled which looks quite weak. I d like to see something solid which doesn t look like it will slide back if the device is treated as roughly as a phone. I think that the lack of a solid lens cover could be the one factor that prevents it from being used as a replacement for a smart phone. Apart from that a Galaxy Camera and a cheap GSM phone could perform all the functions of a high end phone such as the Galaxy S3 while also producing great pictures. It would probably make sense for retailers to bundle a cheap phone with a Galaxy Camera for this purpose. Tablet Features The device boasts WiFi Direct to allow multiple cameras and phones to talk to each other without a regular WiFi access point [3]. I didn t test this and I don t think it would be particularly useful to me, but it seems like a handy feature for less technical users. It can connect to the Internet via Wifi or 3G, supports automatic upload of pictures (it comes with Dropbox support by default like the Galaxy S3), and has a suite of photo and video editing software. I don t expect that any photo editing software that runs on an Android device would be much good (I think that you really need fine cursor control with a mouse and a high resolution screen), but it would probably be handy for sending out a first draft of photos. Most Android apps should just work, the exceptions being apps that rely on a camera that faces the user or full phone functionality. So the Galaxy Camera can do almost anything that an Android phone or tablet can do. Value The RRP for the Galaxy Camera is $599, that puts it in the same price range as a DSLR with a single lens. While that s not a bad price when compared to smart-phones (it s cheaper than the LTE version of the Galaxy S3 phone) it s still quite expensive for a camera that s not a DSLR. Fortunately Kogan is selling it for $469 and has free shipping at the moment [4]. This still makes it more expensive than some of Bridge Cameras which probably have significantly better optical features, but in terms of what the typical user can do with a camera the Galaxy Camera will probably give a much better result. The sensor in the Galaxy Camera is smaller than that in the Nokia 808 PureView [5] (1/2.3 vs 1/1.2 ) so the Nokia PureView should be able to take better pictures in some situations. Unfortunately the Nokia 808 doesn t run Android, I d probably own one if it did. Some of the reviews are rather harsh, the Verge has a harsh but fair review by Aaron Souppouris which makes a number of negative comparisons to cheaper cameras [6]. I really recommend reading Aaron s review as there s a lot of good information there. But I think that Aaron is missing some things, for example he criticises the inclusion of ebook software by saying that he wouldn t read a book on a camera. But the device is a small tablet computer which also has a compact camera included. I can easily imagine someone reading a book or playing Angry Birds on their camera/tablet while in transit to where they are will photograph something. I can also imagine a Galaxy Camera being a valuable tool for a journalist who wants to be able to write short articles and upload pictures and video when out of the office. Aaron concludes by suggesting that the Galaxy Camera is a $200 camera with $300 of editing features. I think of it as $200 in camera hardware with software that allows less skilled users to take full advantage of the hardware and the ability to do all the software/Internet things that you would do on a $450+ smart-phone. Would I Buy One? No. The Galaxy Camera is among other things a great toy, I d love to have one to play with but I can t spare $469 on one. Part of the reason for this is that my wife just bought a DSLR and is getting lessons from a professional photographer, so I really won t get better pictures from a Galaxy Camera. The DSLR on auto mode will allow me to take pictures that will usually be better than a Galaxy Camera can achieve (sometimes you just can t beat a good lens). For more demanding pictures my wife can tweak the DSLR. The 120fps video is a really nice feature, I don t know if my wife s DSLR can do that, but it s a toy feature not something I really need. I ve just bought a Galaxy S3 which is a great little tablet computer (most of the time it won t be used for phone calls). I don t need another 4.8 tablet so a significant part of the use of the Galaxy Camera doesn t apply to me. I recommend the Galaxy Camera to anyone who wants to take good photos but can t get a DSLR and lessons on how to use it properly. But if you would rather get a 35mm camera with interchangeable lenses that runs Android then it might be worth waiting. I expect that the Galaxy Camera will be a great success in the market (it s something you will love when you see it). That will drive the development of similar products, if Samsung doesn t release a 35mm Android camera soon then someone else will (for example Sony develops both high end cameras and Android phones). If my wife didn t have a DSLR then I d probably have bought a Galaxy Camera already. I will recommend it to my parents and many other people I know who want an OK camera and can benefit from a tablet, but don t know how to use a DSLR properly (or don t want to carry a bulky camera).

• [1] http://en.wikipedia.org/wiki/Galaxy_Camera
• [2] http://etbe.coker.com.au/2011/08/24/digital-cameras/
• [3] http://en.wikipedia.org/wiki/WiFi_Direct
• [4] http://www.kogan.com/au/buy/samsung-galaxy-camera-ek-gc100-white/
• [5] http://en.wikipedia.org/wiki/Nokia_808
• [6] http://www.theverge.com/2012/12/3/3706206/samsung-galaxy-camera-review

A new version of the digest package (which generates hash function summaries for arbitrary (and possibly nested) R objects using any of the standard md5, sha-1, sha-256, sha-512 or crc32 algorithms) is now on CRAN, and I will push the corresponding version into Debian in a moment. For this release, Hannes Muehleisen added support for sha-512 using an older standalone function by Aaron D. Gifford which I had to whip into slightly more portable shape to work on Windows. (Hint: uint32_t from stdint.h, not u_int32_t) CRANberries provides the usual summary of changes to version 0.5.2. As usual, our package is available via the R-Forge page leading to svn and tarball access, my digest page and the local directory here.

Powell's was holding an anniversary sale and giving out free gift cards with a qualifying order, so it seemed like a good idea to pick up a few books. Ben Aaronovitch Midnight Riot (sff)
Patricia Briggs Alpha & Omega (sff)
Patricia Briggs Moon Called (sff)
Mark Forster Get Everything Done (non-fiction)
Lynn Galli Wasted Heart (romance)
Sam Harris Lying (non-fiction)
K.E. Lane And Playing the Role of Herself (romance)
Jack McDevitt Firebird (sff)
Marshall Rosenberg Nonviolent Communication (non-fiction)
Deborah Tannen The Argument Culture (non-fiction)
Bill Willingham, et al. Fables: Animal Farm (graphic novel) I'm not sure if the Forster will be worth the cost, but I've really liked his other writing about time management, so I thought I'd take a chance on it. I think the Tannen and the Rosenberg are on roughly the same topic, but I arrived at them through different recommendation paths. I've been feeling like reading about how to communicate lately. The McDevitt novel is the latest in the Alex Benedict series. I'm only a couple of books away from being caught up. I've gotten lots of positive recommendations for Aaronovitch's series, so thought I'd get the first one. There's also a variety of random stuff, mostly on the Kindle, that I'd not added to one of these posts, noted here mostly for my own records. Alpha & Omega is a novella rather than a novel, which I grabbed because I picked up and read Cry Wolf during vacation.

Review: Set This House in Order, by Matt Ruff

Publisher:	Perennial
Copyright:	2003
Printing:	2004
ISBN:	0-06-095485-X
Format:	Trade paperback
Pages:	479

A textbook case history would say that Andy Gage has disassociative identity disorder (referred to here as the more common although medically inaccurate "multiple personality disorder" for reasons of clarity and to match casual speech, according to the author). However, that's a completely inadequate description. A better description would be that Andy Gage's body is inhabited by a large set of different personalities, who take turns being in control. The original Andy Gage is dead, shattered by horrific childhood abuse. Aaron became the dominant personality and gave structure to the internal geography of the mind they all share, with the help of extensive counselling. But Aaron was exhausted by being the public face and maintaining control of the shared body. Hence, Andrew. Andrew Gage is the first-person narrator of half of this book. He is twenty-eight, the same age as the body, but he's only existed for two years. He considers Aaron his father, since Aaron called him out of the lake inside their internal metal geography. His purpose is to run the body and be the public face of the family. He is by turns capable and naive, shy on personal experience, but able to get the help of the others. The one heard from most frequently is Adam, a rude and mischevious teenager who also has an uncanny ability to read other people, and who frequently watches through the body's eyes from what the family calls the "pulpit" and gives Andrew advice. Andrew works for Julie Sivik, a woman he met the day he quit his job at a big-box store because he couldn't explain his behavior to his boss without discussing being a multiple. Julie is a wholly impractical, slightly unethical, well-meaning force of chaos who serves as the thread that draws the plot of the book together. She's running a startup on ingenuity and borrowed money that is attempting to create a comprehensive and customizable virtual reality world. She's the one who finds Penny and hires her, in part for her skills, and in part to push Penny and Andrew together. Penny Driver has the same condition as Andrew, but Penny is only vaguely aware of it. The world for her is full of blackouts, lost time, waking up in strange and sometimes horrific situations, and attempting to structure her life based on notes left for herself that she didn't write. When she meets Andrew, she's barely making it through life; thankfully, one of her personalities is remarkably good at computers, and that has been enough to make a living despite the chaos. But Julie is the unintentional catalyst for her losing that job, the best job she's ever had, leaving her with no real alternative but to take the job with Julie's company. There are a few things that are important to say up-front about this book. First, neither I nor the author are actually multiples (or, put another way, neither of us have DID), and writing about atypical mental states is fraught with peril. It's very easy to misrepresent other people's experiences, and multiple personalities in particular are frequent material for dramas of all sorts. There's something people find inherently fascinating about the idea, and it's easy for that fascination to run away with reality and distort the lived experience. I recommend reading Ruff's FAQ about the book, which spells out what his goals are and what approach he took. I did some quick searches and didn't uncover a lot of anger or dislike of the book among either people who have DID or psychiatrists, and it seems true to the descriptions of internal reality I've heard from people who are multiples. But despite being more respectful of real experience than most fictional treatments of DID, this is still a work of fiction written by someone without first-hand experience, and it's worth keeping that in mind. Second, DID is an intensely controversial diagnosis among psychiatrists. There are, generally speaking, two major controversies. One is over whether or not DID can actually be caused by traumatic experiences such as childhood abuse, or whether it is a creation of psychiatrists, a mental model that they've imposed on their patients or convinced their patients to believe. Set This House in Order comes down firmly on the first side of that controversy: that DID can be created by abuse and is not a creation of psychotherapy. (So, I should note, does every multiple that I've ever talked to or read.) Second, there is significant controversy over treatment, with some psychiatrists believing that integration of the personalities into one is the only true treatment, and others believing that multiples can remain stable and functional as multiple personalities. Set This House in Order leans strongly towards the view that multiples can be stable as multiples and that while integration may be appropriate for some, it's not for all. Both of these viewpoints are presented very convincingly by the narrative in the book, and both are consistent with the beliefs of a lot of multiples, but it's worth remembering that these are active and often quite heated controversies and a novel is not evidence. It's good to be aware of both of those points, and to read Put This House in Order as something of a what-if, where much more is clear about DID than is actually clear in our world. But once one has done that, one is in for an excellent book. I loved this book from start to finish. (Well, okay, I felt some surprise at the nature of the ending, but it grew on me after further consideration.) It's simply a brilliant novel, with a little bit of everything: very good characterization, messy love affairs, the satisfying tension of someone overcoming great odds to put themselves together with the help of friends, a startlingly original perspective of the world, some excellent surprises, and some absolutely wonderful characters. Andrew is a good choice of narrator: he brings an earnest calm to a story that could otherwise be quite chaotic and provides a great vantage point for the reader. And I found Penny (and particularly some of her other souls) impossible not to like. Set This House in Order is a tense book in places, and it's a book where people's lives break down before they can get better. It's also a book about rather horrific abuse, and while nearly all of the specifics happen off-stage, imagination fills in some nasty details. But Ruff manages to write it with a structure and with enough calm assurance that I felt just reassured enough without taking away the tension. That's a tricky balance; few books do it better. And the supporting characters are a delight. One might think, with all the complexity happening for both Andrew and Penny, that Ruff wouldn't have room in the narrative for other characters, but one would be wrong. There are very few saints and demons here, just a lot of complex people, with their own problems and their own perspectives, who are generally trying to do the best they can from day to day. There are some clear-cut villains in the past Ruff wisely makes no attempt to add shades of grey to abusers but even there one can see some clear signs of untreated severe mental illness and chaotic lives rather than mustache-twirling pre-meditated villainy. This is, in short, a book full of people, in all their messy complications, which is what makes it so delightful. I'm not going to say anything more about the plot, or even the various personalities, since discovering the direction of the book was so much of the fun. I will say that, despite the subject matter and some of the events, the book is surprisingly gentle. One of the things about it that I loved the most was the emotional tone Ruff gave to being a multiple: the sense of a squabbling but tight-knit family looking out for each other. I can't judge the reality of this impression, but it makes for a wonderful story. Highly recommended. The best book I've read so far this year. Rating: 10 out of 10

DPL August report, posted on d-d-a a while ago (yep, I forgot to blog it up to now!, sorry for the oldies).

---

Dear project members, August has been a month with a good deal of vacations for many of us, including yours truly. Therefore the monthly report of DPL activities will be briefer than usual. Which is good, as it'll leave all my readers more time to do NMUs and fix RC bugs! Ongoing discussions

• Discussion about the new trademark policy has continued on -project. I've collected quite some feedback and summarized it in a request for change that I've passed on to SFLC. I'm waiting for a second draft from them, before reviving the thread.
• on the long-running issue of de-duplicating DebConf infrastructure wrt Debian's, I've been happy to support the proposal of moving DebConf's list to Debian's software and infrastructure. There seems to be interest and consensus, but the DebConf team could probably use a couple more helping hands to make it real
• starting from the practical need of dealing with minified JavaScript files, an interesting discussion sparkled on -devel. It quickly turned in both musing on the notion of archive-wide DFSG-freeness and some new devscript tooling by Andreas Tille. AFAIU some extra testing would be welcome

Assets

• The DFGS-free logo relicensing I've mentioned last month is now on its way. Following advice from SFLC, I've slightly patched the current trademark policy (mainly to clarify scope), just enough to decouple logo relicensing under LGPLv3/CC-BY-SA 3.0 from the finalization of a new trademark policy (see above). I've also verified that the license choice is fine with teams that regularly deal with the logos. Ball is now in SPI camp (as copyright holder). They should finalize the relicensing during the forthcoming board meeting, on September 13th.
• We heard back from the trademark office in Japan, who accepted our request to transfer it from local Debian contributors to SPI. That being mainly a bureaucratic acknowledgement, we'll now need to way for a while until they confirm the transfer has been finalized. My eternal thanks for help with this matter goes to Kenshi Muto.

Core teams

• Manoj Srivastava has stepped down as tech-ctte member. I hereby thank him for his long and valuable service in that capacity. The tech-ctte has now one vacant seat, and I've encouraged present tech-ctte members to propose to me a suitable candidate. In an attempt to promote diversity of views within the tech-ctte, they are now considering various ways to solicit candidatures. You'll hear from them about this soon, I suspect.
• In the meantime, monthly IRC meetings of the tech-ctte are continuing, as you might have noticed from tech-ctte decision reports posted on this list. As it seemed to be lingering, at the last meeting I've solicited progress on the Python maintenance issue.
• The auditors have noticed the unclear situation of the Policy Editor delegation, which seems to rely on a "base" delegation dating back to Antony Towns' office and various subsequent delegation "patches". After discussion with Russ Allbery I've then proceeded to a public "ping" to assess the interest of current editors in continuing their work in such capacity, and encourage new volunteers to apply as editors. Based on the feedback I've got, I'm now preparing a new Policy Editor delegation.

Legal and RC fun

• Solicited by Jonathan Nieder and Helge Kreutzmann, I've looked into #555168 and sought legal advice to answer various questions related to the non-copyrightability of locale data. Aaron Williamson of SFLC has been blazing fast in answering them (thanks!), and Jonathan and Helge has now all the needed information to help addressing that (RC) issue. Keep an eye on the bug log for more details.

Hardware

• I've approved buying 6 console servers to improve DSA ability to act on remote machines, without having to bother^Wrely on local admins when not strictly needed. ~550 USD well spent.

See? It's been quick(er)! Talk to you here next month, with a much lower count of Wheezy RC Bugs on the horizon, hopefully.
Cheers.

---

PS the boring day-to-day activity log for August 2012 is available at master:/srv/leader/news/bits-from-the-DPL.txt.201208